Reproducible research
with Nix and rix

Short intro to the workshop



Bruno Rodrigues
Intro: Jürgen Schneider

20 March 2024

Aktuelle Herausforderungen

Wann bereitstellen, wann archivieren?


Kosten:

  • Ressourcen für Aufbereitung
  • Erschwert/Verunmöglicht Primärforschung?

Nutzen:

  • Analysepotential
    • Umfang Datenmaterial
    • Repräsentativ oder spezifisch (“seltene Daten”)
  • Zusätzliche Daten (z.B. Leistungsdaten, Interviews)

Housekeeping

Please check:

rix installed

Nix installed (if not, we’ll do an alternative using github actions)


Alternative to installation of Nix
Do you have git installed?

Do you have a github account?

The DIPF Open Science Code

  • event series “Open Science - Opportunities for Educational Research”
  • DIPF working group “Open Research and Practice” is organising
  • See further events on https://www.leibniz-openscience.de/event-calendar/
  • the code
  • pyramid “make it easy” (how this workshop fits into the greater idea of …)

The challenge

  • You want to redo parts of the analyses: Because a reviewer is interested in …, because you want to report certain group comparisons on a conference. But when you re-run your code, you get error messages
  • A reviewer asks you to share your data so that s/he can check your analyses

The challenge

Compuational Reproducibility: “In principle, all reported evidence should be reproducible. If someone applies the same analysis to the same data, the same result should occur.” (Nosek et al., 2022, p. 721)

(Crüwell et al., 2023)

Investigated all articles from one issue in Psychological Science: 1 exactly reproducible, 3 essentially reproducible after minor deviations

“Code-specific issues include (a) a lack of shared analysis code or modeling code and (b) issues with package or software versions (often resolvable but sometimes only with considerable effort).”

(Hardwicke et al., 2021) Investigated all articles with open data badges from Psychological Science in 2014-2015:

all target values in 9 out of the 25 articles (36%, CI [19,57]) were reproducible, with the remaining 16 articles (64%, CI [43,81]) containing at least one major numerical discrepancy. After requesting input from original authors, several issues were resolved through the provision of additional information and ultimately all target values in 15 (60%, CI [39,78]) articles were reproducible, with the remaining 10 (40%, CI [22,61]) articles containing at least one major numerical discrepancy. (p. 4)

(Obels et al., 2020) analyzed 62 registered reports. 36 shared both, code and data.
computationally reproduced (21 out of 36, or 58%)

The challenge

  • Artner slides

Reasons

  • copy paste errors
  • … (see slides Metascience)
  • Package conflicts
  • Non-standardized computational environment

Input-Output-Documents

renv

Containers: Holepunch

https://doi.org/10.1177/2378023119849803 https://doi.org/10.15626/MP.2018.892

Recording

Vielen Dank



Jürgen Schneider

References

Crüwell, S., Apthorp, D., Baker, B. J., Colling, L., Elson, M., Geiger, S. J., Lobentanzer, S., Monéger, J., Patterson, A., Schwarzkopf, D. S., Zaneva, M., & Brown, N. J. L. (2023). What’s in a Badge? A Computational Reproducibility Investigation of the Open Data Badge Policy in One Issue of Psychological Science. Psychological Science, 34(4), 512–522. https://doi.org/10.1177/09567976221140828
Hardwicke, T. E., Bohn, M., MacDonald, K., Hembacher, E., Nuijten, M. B., Peloquin, B. N., deMayo, B. E., Long, B., Yoon, E. J., & Frank, M. C. (2021). Analytic reproducibility in articles receiving open data badges at the journal Psychological Science : An observational study. Royal Society Open Science, 8(1), 201494. https://doi.org/10.1098/rsos.201494
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., Rohrer, J. M., Romero, F., Scheel, A. M., Scherer, L. D., Schönbrodt, F. D., & Vazire, S. (2022). Replicability, Robustness, and Reproducibility in Psychological Science. Annual Review of Psychology, 73(1), 719–748. https://doi.org/10.1146/annurev-psych-020821-114157
Obels, P., Lakens, D., Coles, N. A., Gottfried, J., & Green, S. A. (2020). Analysis of Open Data and Computational Reproducibility in Registered Reports in Psychology. Advances in Methods and Practices in Psychological Science, 3(2), 229–237. https://doi.org/10.1177/2515245920918872

Credit

Title page Thomas William on Unsplash

Icons by Font Awesome CC BY 4.0

Slides: bit.ly/shared-slides